Imaging system and method

ABSTRACT

An imaging system includes an image sensor and an image signal processor (ISP). The image sensor generates image data including a set of pixel values. The ISP defines a first subset of pixel values from the set of pixel values. The first subset of pixel values corresponds to at least one region of interest. The ISP defines a second subset of pixel values that is complementary to the first subset of pixel values. The ISP generates a first sub-image based on the second subset of pixel values. The ISP processes the first subset of pixel values to generate a second sub-image. Processing the first subset of pixel values includes at least one of changing a color of one or more pixel values from the first subset of pixel values and scaling the first subset of pixel values. The ISP merges the first and second sub-images to generate an output image.

TECHNICAL FIELD

The present disclosure relates generally to imaging systems and methods,and more specifically to imaging systems and methods for vehicles.

BACKGROUND

Vehicles may include one or more cameras and one or more displays todisplay areas that are not generally visible to a driver, for example,areas to the rear of the vehicle or in a blindspot. An image captured bya camera may include additional information for alerting the driver.Such information is often overlaid on the image after the image has beenprocessed and generated by an image processor. One common disadvantageof overlaying an image is an increase in additional processing. Thisadditional processing for overlaying information on the image mayincrease processing time and computing requirements.

SUMMARY

In one aspect, the present disclosure provides an imaging system for avehicle. The imaging system includes an image sensor, an image signalprocessor (ISP) and a display device. The image sensor is disposed onthe vehicle. The image sensor is configured to generate image data. Theimage data includes a set of pixel values. The ISP is communicablycoupled to the image sensor. The ISP is configured to receive the imagedata from the image sensor. The ISP is further configured to define afirst subset of pixel values from the set of pixel values. The firstsubset of pixel values corresponds to at least one region of interest inthe image data. The ISP is further configured to define a second subsetof pixel values from the set of pixel values. The second subset of pixelvalues is complementary to the first subset of pixel values. The ISP isfurther configured to generate a first sub-image based on the secondsubset of pixel values. The ISP is further configured to process thefirst subset of pixel values to generate a second sub-image. Processingthe first subset of pixel values includes at least one of: (i) changinga color of one or more pixel values from the first subset of pixelvalues; and (ii) scaling the first subset of pixel values. The ISP isfurther configured to merge the first and second sub-images to generatean output image. The display device is configured to display the outputimage received from the ISP.

In an example, the at least one region of interest may correspond to oneor more distance lines with respect to the vehicle.

In an example, the at least one region of interest may include at leastone of an object or an icon.

In an example, the processing of the first subset of pixel values mayinclude changing a color of each pixel value of the first subset ofpixel values.

In an example, the processing of the first subset of pixel values mayfurther include filtering out at least one of red, green and blue colorsfrom each pixel value of the first subset of pixel values.

In an example, the processing of the first subset of pixel values mayinclude magnifying the first subset of pixel values.

In an example, the ISP may be further configured to perform high dynamicrange (HDR) processing of the first subset of pixel values.

In an example, the ISP may be further configured to perform high dynamicrange (HDR) processing of the second subset of pixel values.

In another aspect, the present disclosure provides an imaging method fora vehicle. The method includes receiving image data from an image sensordisposed on the vehicle. The image data includes a set of pixel values.The method further includes defining a first subset of pixel values fromthe set of pixel values. The first subset of pixel values corresponds toat least one region of interest in the image data. The method furtherincludes defining a second subset of pixel values from the set of pixelvalues. The second subset of pixel values is complementary to the firstsubset of pixel values. The method further includes generating a firstsub-image based on the second subset of pixel values. The method furtherincludes processing the first subset of pixel values to generate asecond sub-image. Processing the first subset of pixel values includesat least one of: (i) changing a color of one or more pixel values fromthe first subset of pixel values; and (ii) scaling the first subset ofpixel values. The method further includes merging the first and secondsub-images to generate an output image. The method further includesdisplaying the output by a display device disposed on the vehicle.

In an example, the at least one region of interest may correspond to oneor more distance lines with respect to the vehicle.

In an example, the at least one region of interest may include at leastone of an object or an icon.

In an example, the processing of the first subset of pixel values mayinclude changing a color of each pixel value of the first subset ofpixel values.

In an example, the processing of the first subset of pixel values mayinclude magnifying the first subset of pixel values.

In an example, the imaging method may further include performing highdynamic range (HDR) processing of the first subset of pixel values.

In an example, the imaging method may further include performing highdynamic range (HDR) processing of the second subset of pixel values.

In one aspect, the present disclosure provides an imaging system for avehicle. The imaging system includes an image sensor, and an imagesignal processor (ISP) and a display device. The image sensor isdisposed on the vehicle. The image sensor is configured to generateimage data. The image data includes a set of pixel values. The ISP iscommunicably coupled to the image sensor. The ISP is configured toreceive the image data from the image sensor. The ISP is furtherconfigured to define a first subset of pixel values from the set ofpixel values. The first subset of pixel values corresponds to aplurality of distance lines with respect to the vehicle. The ISP isfurther configured to define a second subset of pixel values from theset of pixel values. The second subset of pixel values is complementaryto the first subset of pixel values. The ISP is further configured togenerate a first sub-image based on the second subset of pixel values.The ISP is further configured to process the first subset of pixelvalues to generate a second sub-image. Processing the first subset ofpixel values includes at least one of: (i) changing a color of one ormore pixel values from the first subset of pixel values; and (ii)scaling the first subset of pixel values. The ISP is further configuredto merge the first and second sub-images to generate an output image.The display device is configured to display the output image receivedfrom the ISP.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description,will be better understood when read in conjunction with the appendeddrawings. For the purpose of illustration, certain examples of thepresent description are shown in the drawings. It should be understood,however, that the disclosure is not limited to the precise arrangementsand instrumentalities shown. The accompanying drawings, which areincorporated in and constitute a part of this specification, illustratean implementation of system, apparatuses, and methods consistent withthe present description and, together with the description, serve toexplain advantages and principles consistent with the disclosure. Thefigures are not necessarily drawn to scale. Like numbers used in thefigures refer to like components. However, it will be understood thatthe use of a number to refer to a component in a given figure is notintended to limit the component in another figure labeled with the samenumber.

FIG. 1 is a schematic side view of an example of a vehicle;

FIG. 2 is a block diagram of an example of an imaging system for avehicle;

FIG. 3 is flow diagram of an example process for image processing;

FIGS. 4A, 4B, 4C, 4D, and 4E illustrate an example of processing ofimage data;

FIGS. 5A and 5B illustrate an example of magnifying a region of interestin image data;

FIGS. 6A and 6B illustrate examples of changing a color of a region ofinterest in image data;

FIGS. 7 and 8 illustrate an example of output images with regions ofinterest;

FIG. 9 illustrates an example of an image without any region ofinterest;

FIGS. 10, 11, and 12 illustrate an example of output images with regionsof interest;

FIGS. 13A and 13B illustrate an example of a change in color of a regionof interest; and

FIG. 14 is a flow chart illustrating an example of an imaging method fora vehicle.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. Accordingly, various changes,modifications, and equivalents of the systems, apparatuses and/ormethods described herein will be suggested to those of ordinary skill inthe art. Also, descriptions of well-known functions and constructionsmay be omitted for increased clarity and conciseness.

It is to be understood that the phraseology and terminology employedherein are for the purpose of description and should not be regarded aslimiting. For example, the use of a singular term, such as, “a” is notintended as limiting of the number of items. Also the use of relationalterms, such as but not limited to, “top,” “bottom,” “left,” “right,”“upper,” “lower,” “down,” “up,” “side,” are used in the description forclarity and are not intended to limit the scope of the disclosure or theappended claims. Further, it should be understood that any one of thefeatures can be used separately or in combination with other features.Other systems, methods, features, and advantages of the disclosure willbe or become apparent to one with skill in the art upon examination ofthe detailed description. It is intended that all such additionalsystems, methods, features, and advantages be included within thisdescription, be within the scope of the present disclosure, and beprotected by the accompanying claims.

Referring now to the Figures, FIG. 1 illustrates an example of a vehicle100. A camera 102 is disposed on the vehicle 100. The camera 102 mayprovide images from all angles around the vehicle 100 depending on therequirements of a driver or an occupant of the vehicle 100. For example,the camera 102 may help the driver to view areas rear of the vehicle100. Further, the camera 102 may also be used to view areas to the sideof the vehicle 100. The camera 102 may also enable the driver to viewareas in a blindspot. The camera 102 may be provided at any location onthe vehicle 100, for example, a rear of the vehicle 100, a side of thevehicle 100, and so forth. The camera 102 may be fixedly or adjustablymounted on the vehicle 100. The camera 102 may be configured to acquireboth still images and moving images (e.g., video). Further, the camera102 may be a digital camera. Though one camera 102 is shown in FIG. 1,multiple cameras may be disposed at different locations on the vehicle100. Although the vehicle 100 is illustrated as a passenger car in FIG.1, the camera 102 may be used with other types of vehicles, for example,buses, trucks, off-road vehicles, motorcycles, aircrafts, bicycles,trams, locomotives, heavy-duty vehicles used in construction andearthworking, and so forth.

FIG. 2 illustrates an imaging system 200 for a vehicle such as thevehicle 100 described in FIG. 1. Referring to FIGS. 1 and 2, the imagingsystem 200 includes the camera 102 and a display device 202. The camera102 includes an image sensor 204 and an image signal processor (ISP)206. Though one image sensor 204 is illustrated in FIG. 2, the camera102 may include multiple image sensors. The camera 102 may furtherinclude a lens (not shown). The image sensor 204 is disposed on thevehicle 100 and configured to generate image data. The ISP 206 iscommunicably coupled to the image sensor 204. The ISP 206 may beconnected to the image sensor 204 by wired connections, wirelessconnections, or a combination thereof. The display device 202 may bedisposed on the vehicle 100 such as in a rear view device and iscommunicably coupled to the ISP 206. The display device 202 may beconnected to the ISP 206 by wired connections, wireless connections, ora combination thereof. Though in the illustrated example of FIG. 2 theISP 206 is shown to be part of the camera 102, in alternative examples,the ISP 206 may be part of a separate processing device which iscommunicably coupled to one or more image sensors. The imaging system200, as shown in FIG. 2, is exemplary in nature. In other embodiments,the imaging system 200 may include multiple cameras and/or multipledisplay devices. In some cases, the display device 202 maysimultaneously or selectively display multiple images from multiplecameras.

The image sensor 204 may be configured to capture and convert light intoelectrical signals. For example, the image sensor 204 may include a CMOSimage sensor (e.g., a CMOS active-pixel sensor (APS)) or a CCD(charge-coupled device) sensor. Generally, the image sensor 204 of thecamera 102 includes an integrated circuit having an array of pixels,where each pixel includes a photodetector for sensing light. The imagesensor 204 may further include a color filter array (CFA) that mayoverlay or be disposed over the pixel array of the image sensor tocapture color information. The color filter array may include a Bayercolor filter array. The color filter array may include an array of smallcolor filters, each of which may overlap a respective pixel of the imagesensor and filter the captured light by wavelength. When used inconjunction, the color filter array and the photodetectors may provideboth wavelength and intensity information with regard to light capturedthrough the camera 102, which may be representative of a captured image.

The ISP 206 may provide for various image processing steps, such asdefective pixel detection/correction, lens shading correction,demosaicing, high dynamic range (HDR) processing, image sharpening,noise reduction, gamma correction, image enhancement, color-spaceconversion, image compression, chroma sub-sampling, color shifting, edgeenhancement, image scaling operations, other types of pixelmanipulation, and so forth. In some examples, the ISP 206 may includevarious subcomponents and/or discrete units of logic that collectivelyform an image processing pipeline for performing each of the variousimage processing steps. These subcomponents may be implemented usinghardware (e.g., one or more processors) or software, or via acombination of hardware and software components. The processor(s) of theISP 206 may include one or more microprocessors, such as one or moregeneral-purpose microprocessors, one or more special-purposemicroprocessors and/or application-specific microprocessors (ASICs), ora combination of such processing components. The instructions or data tobe processed by the processor(s) may be stored in a computer-readablemedium, such as a memory device. The memory device may be provided as avolatile memory, such as random access memory (RAM) or as a non-volatilememory, such as read-only memory (ROM), or as a combination of one ormore RAM and ROM devices. The memory may store a variety of informationand may be used for various purposes. For example, the memory may storefirmware for the ISP 206 and the camera 102, such as a basicinput/output system (BIOS), an operating system, various programs,applications, or any other routines that may be executed on the camera102, including user interface functions, processor functions, and soforth. In addition, the memory may be used for buffering or cachingduring operation of the ISP 206 and the camera 102. In an example, theISP 206 may be implemented in an integrated circuit, such as a system onchip (SoC).

The display device 202 may be configured to display an output imagereceived from the ISP 206. The display device 202 may include any typeof device including a display, for example, but not limited to, adisplay in an instrument panel of the vehicle 100, a head-up display(HUD), a smartphone, a tablet computer, a rearview or a sideview mirrorincluding a display, and so forth. The display device 202 may include aliquid crystal display (LCD), a light-emitting diode (LED) display, acathode ray tube (CRT) display, a plasma display panel (PDP), anelectrolumiscent display (ELD), and so forth. Further, the displaydevice 202 may be touch-enabled. The display device 202 may be fixedlyor adjustably mounted on the vehicle 100. The display device 202 may belocated such that the driver can view the output image displayed on thedisplay device 202. The display device 202 may be disposed internally orexternally with respect to the vehicle 100. Though one display device202 is shown in FIG. 2, in other examples, the ISP 206 may be coupledwith multiple display devices.

The imaging system 200 may allow the driver to view an object 104located behind and/or to a side of the vehicle 100. The imaging system200 may also process the output image to highlight one or more regionsof interest in the output image. For example, the imaging system 200 mayindicate one or more distance lines with respect to the vehicle 100. Adistance line is a virtual line that indicates a distance from thevehicle 100. A distance line may be disposed to a rear, a front or aside of the vehicle 100. The distance may be measured from any referencepoint of the vehicle 100, for example, a rear end of the vehicle 100.Referring to FIG. 1, the imaging system 200 may indicate distance linesDL1, DL2, DL3 with respect to the vehicle 100 in the output imagedisplayed on the display device 202. In the illustrated example, thedistance lines DL1, DL2, DL3 are located to the rear of the vehicle 100.The distance lines DL1, DL2, DL3 may be located at distances m1, m2, m3,respectively, with respect to the vehicle 100. In the illustratedembodiment, m1<m2<m3. In an example, m1 may be about 3 meters (m), m2may be about 10 m and m3 may be about 25 m. The distance lines DL1, DL2,DL3 may allow the driver to estimate a distance between the vehicle 100and the object 104.

FIG. 3 shows a flow diagram 300 for an example of a process for imageprocessing implemented by the imaging system 200. Referring to FIGS.1-3, at step 302, an image sensor 204 of the camera 102 generates imagedata 304. At step 305, the image data 304 is transmitted to the ISP 206for processing. The image data 304 may include a set of pixel valuescorresponding to the array of pixels of the image sensor 204. The imagedata 304 may include a raw or unprocessed data bit stream of the pixelvalues. In some cases, the pixel values may include RGB (red, green,blue) data. FIG. 4A illustrates the image data 304 including a set ofpixel values ST. In this example, the set of pixel values ST include oneor more pixel values 402. In the illustrated example of FIG. 4A, thepixel values 402 are arranged in a 12×16 array. However, a number ofrows and columns of the pixel values 402 may vary based on theconfiguration of the image sensor 204. Each pixel value 402 may includecolor data (e.g., RGB data) and intensity data. The image data 304 maydynamically change based on movement of the vehicle 100 and change inthe surroundings of the vehicle 100.

Referring back to FIG. 3, at step 306, the ISP 206 receives the imagedata 304 from the image sensor 204 and processes the image data 304. Inan example, the ISP 206 may define a first subset of pixel values S1from the set of pixel values ST. The first subset of pixel values S1 maycorrespond to at least one region of interest (ROI) in the image data304. For example, the at least one region of interest may correspond toone or more distance lines DL1, DL2, DL3 with respect to the vehicle100. In another example, the at least one region of interest includes atleast one of an object and an icon. As shown in FIG. 4B, the firstsubset of pixel values S1 may correspond to three regions of interest406 in the image data 304. Each region of interest 406 may have anysuitable shape, for example, rectangular, linear, circular, polygonal,elliptical, or any irregular shape. The ISP 206 may identify the regionsof interest 406 based on instructions stored in the associated memory.The regions of interest 406 may be user-defined or defined by amanufacturer. The regions of interest 406 may also dynamically changebased on various parameters, such as level of ambient light (day/night),proximity to an object, speed of the vehicle 100, among other factors. Anumber and shapes of the regions of interest 406 may therefore vary asper application requirements. In some examples, each region of interest406 may correspond to a distance line relative to the vehicle 100. Forexample, the first subset of pixel values S1 may correspond to aplurality of distance lines DL1, DL2, DL3 relative to the vehicle 100.In other examples, each region of interest 406 may include at least oneof an object (e.g., image of the object 104) and an icon.

The one or more regions of interest 406 may correspond to certain rangesof pixel values 402 that represent the distance lines DL1, DL2, DL3 inthe image data 304. For example, the pixel values 402 in the range fromC11 to N11 may represent the distance line DL1 in the array of pixels.Further, the pixel values 402 in the range from C7 to N7 may representthe distance line DL2 in the array of pixels. Moreover, the pixel values402 in the range from C2 to N2 may represent the distance line DL3 inthe array of pixels. The first subset of pixel values S1 may thereforeinclude the pixel values 402 from C11 to N11, from C7 to N7 and from C2to N2.

The ISP 206 may further define a second subset of pixel values S2 fromthe set of pixel values ST. The second subset of pixel values S2 may becomplementary to the first subset of pixel values S1. In other words,the second subset of pixel values S2 may include the pixel values 402 ofthe set of pixel values ST that do not belong to the first subset ofpixel values S1, i.e., S2=ST−S1. FIG. 4C illustrates the first subset ofpixels S1 and the second subset of pixels S2 separated from each other.

In an example, the ISP 206 may process the first subset of pixel valuesS1 and the second subset of pixel values S2 separately. The processingof the first subset of pixel values S1 may be different from theprocessing of the second subset of pixel values S2. Referring back toFIG. 3 together with FIG. 4D, at step 308, the ISP 206 may generate afirst sub-image IM1 using the second subset of pixel values S2. Thefirst sub-image IMI may be a colored image or a greyscale image. The ISP206 may perform various image processing steps on the second subset ofpixel values S2, such as defective pixel detection/correction, lensshading correction, demosaicing, image sharpening, noise reduction,gamma correction, image enhancement, color-space conversion, imagecompression, chroma sub-sampling, among other processing steps. The ISP206 may be further configured to perform high dynamic range (HDR)processing of the second subset of pixel values S2. HDR processing mayprovide a greater dynamic range of luminosity that can be perceived by ahuman eye. In some cases, HDR processing may include tone mapping. TheISP 206 may therefore partially generate an image, i.e., the firstsub-image IMI, by processing the second subset of pixel values S2.

Still referring to FIG. 3 together with FIG. 4D, at step 310, the ISP206 may generate a second sub-image IM2 using the first subset of pixelvalues S1. The second sub-image IM2 may be a colored image or agreyscale image. The ISP 206 may perform various image processing stepson the first subset of pixel values S1, such as defective pixeldetection/correction, lens shading correction, demosaicing, imagesharpening, noise reduction, gamma correction, image enhancement,color-space conversion, image compression, chroma sub-sampling, amongother processing steps. The processing of the first subset of pixelvalues S1 may further include at least one of changing a color of one ormore pixel values 402 from the first subset of pixel values S1 andscaling the first subset of pixel values S1. Processing the first subsetof pixel values S1 may include pixel manipulation in addition to theprocessing carried out on the second subset of pixel values S2. Pixelmanipulation may include color shifting and/or scaling.

In some examples, processing the first subset of pixel values S1 mayfurther include changing a color of each pixel value 402 of the firstsubset of pixel values S1. In further examples, processing the firstsubset of pixel values S1 may also include filtering out at least one ofred, green and blue colors from each pixel value 402 of the first subsetof pixel values S1. Filtering of red, green and/or blue colors may beachieved by applying a color filter on the first subset of pixel valuesS1. Removal of red, green and/or blue colors may highlight the regionsof interest 406 with respect to the adjacent areas in the image data304. The ISP 206 may dynamically change the color of each region ofinterest 406 based on various parameters, such as ambient lightconditions, color of the adjacent areas in the image data 304, speed ofthe vehicle 100, proximity of each region of interest 406 to the vehicle100, among other parameters. For example, the ISP 206 may apply a firstpredefined color shift or change during the day and a second predefinedcolor shift during the night. In another example, the ISP 206 maydetermine a color of each pixel value 402 surrounding each regioninterest of interest 406. The ISP 206 may have chosen a predefinedfiltering process to provide an intended color to each region ofinterest 406. If an intended color of the region of interest 406 issubstantially close to that of the surrounding pixel values 402, the ISP206 may adjust the color of each region of interest 406 so that thedriver can recognize each region of interest 406. For example, the colorof each region of interest 406, may be adjusted to a color belonging tothe same color family in order to distinguish each region of interest406 from surrounding portions. In some embodiments, the regions ofinterest 406 may have different colors.

The first subset of pixel values S1 may be selectively or additionallyscaled. Scaling the first subset of pixel values S1 may be achieved byvarious interpolation techniques, such as nearest-neighborinterpolation, bilinear interpolation, among other interpolationtechniques. The first subset of pixel values S1 may be interpolatedoutwards or inwards. The second subset of pixel values S2 may not bescaled similarly. In some examples, processing the first subset of pixelvalues S1 may further include magnifying the first subset of pixelvalues S1. The magnified regions of interest 406 when merged with therest of the image may result in breaks or discontinuities atcorresponding interfaces. The scaling ratio or amount of magnificationmay depend on various factors and can dynamically change as the factorschange. The factors may include ambient light conditions, speed of thevehicle 100, proximity of each region of interest 406 to the vehicle100, and so forth. For example, magnification may be increased duringlow ambient light conditions (e.g., during the night) as compared togood ambient light conditions (e.g., during the day). In anotherexample, the magnification of each region of interest 406 may changebased on a change in distance between each region of interest 406 andthe vehicle 100. An increase in magnification may easily attract theattention of a driver. In other words, greater magnification may providea more distinct warning to a driver. A region of interest may also bedynamically identified and magnified during an emergency. For example,an object may suddenly appear near the vehicle 100, and thecorresponding region of interest in the image data 304 may be identifiedand magnified to attract the attention of the driver.

The ISP 206 may be further configured to perform high dynamic range(HDR) processing of the first subset of pixel values S1. HDR processingmay provide a greater dynamic range of luminosity that can be perceivedby a human eye. In some cases, HDR processing may include tone mapping.The ISP 206 may therefore partially generate an image, i.e., the secondsub-image IM2, by processing the first subset of pixel values S1. Insome examples, HDR processing of the first subset of pixel values S1 maybe performed first and then pixel manipulation (scaling and/or colorshifting) may be subsequently performed on the HDR processed firstsubset of pixel values S1.

In some examples, the ISP 206 may generate the second sub-image IM2corresponding to all the regions of interest 406. In other examples, theISP 206 may generate separate sub-images for the respective regions ofinterest 406. The processing of the regions of interest 406 may alsovary from each other. For example, the color shifting and/or scaling mayvary across the regions of interest 406.

Referring to FIG. 4E, the ISP 206 may be further configured to merge thefirst and second sub-images IM1, IM2 to generate an output image IMO. Inthis example, the ISP 206 transmits the output image IMO to the displaydevice 202. Referring back to FIG. 3, at step 312, the display device202 displays the output image IMO received from the ISP 206. The mergingof the first and second sub-images IM1, IM2 may be achieved based onlocations of the pixel values 402 in the pixel array. For example, animage portion at B2, as shown in FIG. 4B, may be disposed adjacent to animage portion at C2. In case the regions of interest 406 are magnified,portions of the second sub-image IM2 may be overlaid on portions of thefirst sub-image IM1. Due to magnification, certain portions of thesecond sub-image IM2 may overlap with certain portion of the firstsub-image IM1. In such cases, the portions of the second sub-image IM2may be overlaid on the corresponding portions of the first sub-imageIM1. For example, an image portion corresponding to B2 in the secondsub-image IM2 may be overlaid on a corresponding image portion of thefirst sub-image IM1. Processed pixel values of the second sub-image IM2may overwrite processed pixel values of the first sub-image IM1.

Portions of the output image IMO corresponding to the regions ofinterest 406 may have undergone pixel manipulation that has not beenimplemented in the rest of the output image IMO. The output image IMOmay dynamically change based on the movement of the vehicle 100 andchange in surroundings. In some examples, the ISP 206 may furthergenerate a video output for display at the display device 202.

In the illustrated example of FIG. 3, all the processing of the imagedata 304 may be done by the ISP 206 in a pre-processing stage, i.e., atsteps 306, 308 and 310. Pre-processing may include any activity thatoccurs before an image is generated from raw data. In some examples, nopost-processing may be required on the output image IMO. Post-processingmay include any activity that occurs after the image is generated fromraw data. For example, no overlay may have to be applied on top of theoutput image IMO by post-processing.

FIGS. 5A and 5B illustrate an example of a pixel manipulation method formagnifying a region of interest 502 in image data 500. In this example,image data 500 includes an array of pixel values 504 arranged in a 12×16array. The region of interest 502 is a rectangular region with C3, C5,N3 and N5 as its corners. One or more of the pixel values 504 in theregion of interest 502 may be interpolated outwards for magnifying theregion of interest 502. The ISP 206 (shown in FIG. 3) may perform themagnification. The pixel value 504 at C3 may be interpolated toadjoining pixels, namely, B2, B3 and C2. Similarly, the pixel value 504at C4 may be interpolated to B4. Other pixel values 504 may be similarlyinterpolated and an example of the resulting magnified region 506 isshown in FIG. 5B. The interpolated pixel values 504 may be overlaid onthe surrounding pixel values 504. The interpolated pixel values 504 ofthe magnified region 506 may overwrite the surrounding pixel values 504.Since only the region of interest 502 is magnified and not the otherparts of the image data 500, discontinuities or breaks may be present atone or more interfaces between the magnified region 506 and surroundingportions of the image. For example, a break may be present between rows1 and 2.

FIGS. 6A and 6B illustrate an exemplary color shifting method of aregion of interest 602 in image data 600. The image data 600 includes anarray of pixel values 604 arranged in a 12×16 array. The region ofinterest 602 is a rectangular region with D3, D5, O3 and O5 as itscorners. Color of one or more pixel values 604 in the region of interest602 may be changed by the ISP 206. In the illustrated embodiment of FIG.6B, a color of each pixel value 604 in the region of interest 602 ischanged. The color may be changed by filtering out at least one of red,green and blue colors from each pixel value 604 in the region ofinterest 602. Color shifting may result in a color shifted region 606shown in FIG. 6B.

FIG. 7 illustrates an example of an output image 700 displayed on thedisplay device 202, as previously illustrated in FIG. 2. The outputimage 700 may include one or more regions of interest 702, 704, and 706which correspond to one or more respective distance lines. For example,the one or more regions of interest 702, 704, and 706 may correspond todistance lines DL1, DL2 and DL3, respectively, as described in referencewith FIG. 1. In this example, the regions of interest 702, 704 and 706are magnified with respect to other regions of the output image 700. Dueto magnification, breaks 708, 710, 712 may be disposed between therespective regions of interest 702, 704, 706 and the adjoining portionsof the output image 700 that are not similarly magnified. In an example,the breaks 708, 710, 712 may be easily noticeable by the driver. Usingthe breaks 708, 710, 712, the driver may be able to estimate distancesbetween the vehicle 100 and different objects such as object 104described in reference to FIG. 1.

FIG. 8 illustrates another example of an output image 800 that issimilar to the output image 700. In this example, the three regions ofinterest 702, 704 and 706 in the output image 800 are colored. The pixelvalues in the three regions of interest 702, 704 and 706 have been colorshifted so that three regions of interest 702, 704 and 706 havedifferent colors with respect to the surrounding portions of the outputimage 800. Further, in an example, the three regions of interest 702,704 and 706 may have colors that are different from each other. Thecolors in the three regions of interest 702, 704, 706 may be changed byfiltering out red, green and/or blue colors in the corresponding pixelvalues. The color-coded regions of interest 702, 704 and 706 may provideadditional information to the driver.

FIG. 9 illustrates an example of an image 900 without any region ofinterest. FIG. 10 illustrates an example of an output image 1000 withthree regions of interest 1002, 1004, 1006. Each region of interest1002, 1004, 1006 in this example is a rectangular region correspondingto a respective distance line. For example, the regions of interest1002, 1004, 1006 may correspond to the distance lines DL1, DL2, DL3,respectively, as described in reference with FIG. 1. A color of eachregion of interest 1002, 1004, 1006 may be changed in order to attractthe attention of the driver. The colors of the regions of interest 1002,1004, 1006 may be different from each other. FIG. 11 illustrates anoutput image 1100 including a region of interest 1102. The region ofinterest 1102 is an icon with a substantially triangular shape. Theregion of interest 1102 may be generated by changing the color of thecorresponding pixel values. The icon may provide a warning to thedriver. FIG. 12 illustrates an example of an output image 1200 with aregion of interest 1202. The region of interest 1202 may be an object,such as an obstruction on a road. A color of the region of interest 1202may be changed to alert the driver.

FIGS. 13A and 13B illustrate an example of a change in color of one ormore regions of interest 1302. The region of interest 1302 may be anyone of the regions of interest 1002, 1004, 1004 shown in FIG. 10. Thecolor of the region of interest 1302 has been changed in the example ofFIG. 13B. In some examples, the color of the region of interest 1302 maybe changed in response to an intended or present color of the region ofinterest 1302 being similar to surrounding portions of the image. Thecolor of the region of interest 1302 may be adjusted to a similar color(for example, a different shade of red) or a different color that can berecognized by the driver.

FIG. 14 is a flowchart illustrating an example of an imaging method 1400(hereinafter referred to as “the method 1400”) for a vehicle. The method1400 may be implemented using an imaging system, such as the imagingsystem 200 described above in reference with FIG. 2, for a vehicle, suchas the vehicle 100 described above in reference with FIG. 1. Referringto FIGS. 1-14, at step 1402, the method 1400 includes receiving theimage data 304 from the image sensor 204 disposed on the vehicle 100.The image data 304 includes the set of pixel values ST. The ISP 206receives the image data 304 from the image sensor 204.

At step 1404, the ISP 206 defines a first subset of pixel values S1 fromthe set of pixel values ST. The first subset of pixel values S1 maycorrespond to at least one region of interest in the image data 304. Inan example, the at least one region of interest may correspond to one ormore distance lines DL1, DL2, DL3 with respect to the vehicle 100, asdescribed above in reference with FIG. 1. In another example, the atleast one region of interest includes at least one of an object and anicon.

At step 1406, the ISP 206 may define a second subset of pixel values S2from the set of pixel values ST. The second subset of pixel values S2may be complementary to the first subset of pixel values S1.

At step 1408, the ISP 206 may generate a first sub-image IM1 based onthe second subset of pixel values S2.

At step 1410, the ISP 206 may process the first subset of pixel valuesS1 to generate the second sub-image IM2. Processing the first subset ofpixel values S1 may include at least one of changing a color of one ormore pixel values 402 from the first subset of pixel values S1 andscaling the first subset of pixel values S1. In an example, theprocessing of the first subset of pixel values S1 may include changing acolor of each pixel value 402 of the first subset of pixel values. Inanother example, the processing of the first subset of pixel values S1may include magnifying the first subset of pixel values S1. In someexamples, the method 1400 may further include performing high dynamicrange (HDR) processing of the first subset of pixel values S1. Inadditional examples, the method 1400 may further include performing highdynamic range (HDR) processing of the second subset of pixel values S2.

At step 1412, the ISP 206 may merge the first and second sub-images IM1,IM2 to generate an output image IMO. In an example, portions of thefirst sub-image IM1 may be overlaid or superimposed on correspondingportions of the second sub-image IM2.

At step 1414, the method 1400 may further include displaying the outputimage IMO by the display device 202 disposed on the vehicle 100, asdescribed above in reference with FIGS. 1 and 2. In this example, theISP 206 may transmit the output image IMO to the display device 202, andthe display device 202 may display the output image IMO.

In certain aspects, the imaging system 200 and the method 1400 enableprocessing of one or more regions of interest in image data during thepre-processing stage, i.e., before an output image is generated. In anexample, post-processing of the output image may not be required. Forexample, there is no requirement for applying an overlay on top of theoutput image by post-processing. As a result, processing time andcomputing requirements may be reduced.

Unless otherwise indicated, all numbers expressing feature sizes,amounts, and physical properties used in the specification and claimsare to be understood as being modified by the term “about”. Accordingly,unless indicated to the contrary, the numerical parameters set forth inthe foregoing specification and attached claims are approximations thatcan vary depending upon the desired properties sought to be obtained bythose skilled in the art utilizing the teachings disclosed herein.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat a variety of alternate and/or equivalent implementations can besubstituted for the specific embodiments shown and described withoutdeparting from the scope of the present disclosure. This application isintended to cover any adaptations or variations of the specificembodiments discussed herein. Therefore, it is intended that thisdisclosure be limited only by the claims and the equivalents thereof.

Furthermore, the features of the disclosure disclosed in thisspecification, the claims and the drawings may be employed bothindividually and in any possible combination for practicing thedisclosure in its various exemplary embodiments. In particular, allclaim feature combinations, irrespective of the claim dependencies, arecovered with this application.

LIST OF NUMBERS

-   -   100 Vehicle    -   102 Camera    -   104 Object    -   200 Imaging System    -   202 Display Device    -   204 Image Sensor    -   206 Image Signal Processor (ISP)    -   DL1 Distance Line    -   DL2 Distance Line    -   DL3 Distance Line    -   m1 Distance    -   m2 Distance    -   m3 Distance    -   300 Process Flow    -   302 Block    -   304 Image Data    -   305 Block    -   306 Block    -   308 Block    -   310 Block    -   312 Block    -   402 Pixel Values    -   406 Regions Of Interest    -   ST Set Of Pixel Values    -   S1 First Subset Of Pixel Values    -   S2 Second Subset Of Pixel Values    -   IM1 First Sub-Image    -   IM2 Second Sub-Image    -   IMO Output Image    -   500 Image Data    -   502 Region Of Interest    -   504 Pixel Values    -   506 Magnified Region    -   600 Image Data    -   602 Region Of Interest    -   604 Pixel Values    -   606 Color Shifted Region    -   700 Output Image    -   702 Region of Interest    -   704 Region of Interest    -   706 Region Of Interest    -   708 Break    -   710 Break    -   712 Break    -   800 Output Image    -   900 Image    -   1000 Output Image    -   1002 Region Of Interest    -   1004 Region Of Interest    -   1006 Region Of Interest    -   1100 Output Image    -   1102 Region Of Interest    -   1200 Output Image    -   1202 Region Of Interest    -   1302 Region Of Interest    -   1400 Imaging Method    -   1402 Step    -   1404 Step    -   1406 Step    -   1408 Step    -   1410 Step    -   1412 Step    -   1414 Step

The invention claimed is:
 1. An imaging system for a vehicle, theimaging system comprising: an image sensor disposed on the vehicle andconfigured to generate image data, the image data comprising a set ofpixel values; an image signal processor (ISP) communicably coupled tothe image sensor, the ISP configured to: receive the image data from theimage sensor; define a first subset of pixel values from the set ofpixel values, the first subset of pixel values corresponding to one ormore distance lines that indicate a distance from the vehicle; define asecond subset of pixel values from the set of pixel values, the secondsubset of pixel values being complementary to the first subset of pixelvalues; generate a first sub-image based on the second subset of pixelvalues; process the first subset of pixel values to generate a secondsub-image, wherein processing the first subset of pixel values comprisesat least one of: (i) changing a color of one or more pixel values fromthe first subset of pixel values; and (ii) scaling the first subset ofpixel values; merge the first and second sub-images to generate anoutput image; and a display device configured to display the outputimage received from the ISP, wherein processing the first subset ofpixel values results in an increased visual contrast between the one ormore distance lines and other portions of the output image, whereinprocessing of the first subset of pixel values corresponding to the oneor more distance lines comprises both changing the color of the one ormore distance lines and magnifying the one or more distance lines withrespect to other regions of the output image, wherein the magnifying theone or more distance lines results in breaks and/or discontinuities inthe output image between one or more magnified portions of the outputimage and one or more non-magnified portions of the output image,wherein the first subset of pixel values corresponds to the magnifiedportion of the output image, and wherein the one or more distance linesdynamically change based on movement of the vehicle and/or change in thesurroundings of the vehicle.
 2. The imaging system of claim 1, whereinthe processing of the first subset of pixel values comprises changing acolor of each pixel value of the first subset of pixel values.
 3. Theimaging system of claim 2, wherein the processing of the first subset ofpixel values further comprises filtering out at least one of red, greenand blue colors from each pixel value of the first subset of pixelvalues.
 4. The imaging system of claim 1, wherein the processing of thefirst subset of pixel values comprises magnifying the first subset ofpixel values.
 5. The imaging system of claim 1, wherein the ISP isfurther configured to perform high dynamic range (HDR) processing of thefirst subset of pixel values.
 6. The imaging system of claim 1, whereinthe ISP is further configured to perform high dynamic range (HDR)processing of the second subset of pixel values.
 7. An imaging methodfor a vehicle, the imaging method comprising: receiving image data froman image sensor disposed on the vehicle, the image data comprising a setof pixel values; defining a first subset of pixel values from the set ofpixel values, the first subset of pixel values corresponding to one ormore distance lines that indicate a distance from the vehicle; defininga second subset of pixel values from the set of pixel values, the secondsubset of pixel values being complementary to the first subset of pixelvalues; generating a first sub-image based on the second subset of pixelvalues; processing the first subset of pixel values to generate a secondsub-image, wherein processing the first subset of pixel values comprisesat least one of: (i) changing a color of one or more pixel values fromthe first subset of pixel values; and (ii) scaling the first subset ofpixel values; merging the first and second sub-images to generate anoutput image; and displaying the output image by a display device,wherein processing the first subset of pixel values results in anincreased visual contrast between the one or more distance lines andother portions of the output image, wherein processing of the firstsubset of pixel values corresponding to the one or more distance linescomprises both changing the color of the one or more distance lines andmagnifying the one or more distance lines with respect to other regionsof the output image, wherein the magnifying the one or more distancelines results in breaks and/or discontinuities in the output imagebetween one or more magnified portions of the output image and one ormore non-magnified portions of the output image, wherein the firstsubset of pixel values corresponds to the magnified portion of theoutput image, and wherein the one or more distance lines dynamicallychange based on movement of the vehicle and/or change in thesurroundings of the vehicle.
 8. The imaging method of claim 7, whereinthe processing of the first subset of pixel values comprises changing acolor of each pixel value of the first subset of pixel values.
 9. Theimaging method of claim 7, wherein the processing of the first subset ofpixel values comprises magnifying the first subset of pixel values. 10.The imaging method of claim 7, further comprising performing highdynamic range (HDR) processing of the first subset of pixel values. 11.The imaging method of claim 7, further comprising performing highdynamic range (HDR) processing of the second subset of pixel values. 12.An imaging system for a vehicle, the imaging system comprising: an imagesensor disposed on the vehicle and configured to generate image data,the image data comprising a set of pixel values; an image signalprocessor (ISP) communicably coupled to the image sensor, the ISPconfigured to: receive the image data from the image sensor; define afirst subset of pixel values from the set of pixel values, the firstsubset of pixel values corresponding to a visual icon or identifiedobject; define a second subset of pixel values from the set of pixelvalues, the second subset of pixel values being complementary to thefirst subset of pixel values; generate a first sub-image based on thesecond subset of pixel values; process the first subset of pixel valuesto generate a second sub-image, wherein processing the first subset ofpixel values comprises at least one of: (i) changing a color of one ormore pixel values from the first subset of pixel values; and (ii)scaling the first subset of pixel values; merge the first and secondsub-images to generate an output image; and a display device configuredto display the output image received from the ISP, wherein processingthe first subset of pixel values results in an increased visual contrastbetween the visual icon or identified object and other portions of theoutput image, wherein processing of the first subset of pixel valuescorresponding to the one or more distance lines comprises both changingthe color of the one or more distance lines and magnifying the one ormore distance lines with respect to other regions of the output image,wherein the magnifying the one or more distance lines results in breaksand/or discontinuities in the output image between one or more magnifiedportions of the output image and one or more non-magnified portions ofthe output image, wherein the first subset of pixel values correspondsto the magnified portion of the output image, and wherein the one ormore distance lines dynamically change based on movement of the vehicleand/or change in the surroundings of the vehicle.
 13. The imaging systemof claim 12, wherein the processing of the first subset of pixel valuescomprises changing a color of each pixel value of the first subset ofpixel values.
 14. The imaging system of claim 13, wherein the processingof the first subset of pixel values further comprises filtering out atleast one of red, green and blue colors from each pixel value of thefirst subset of pixel values.
 15. The imaging system of claim 12,wherein the processing of the first subset of pixel values comprisesmagnifying the first subset of pixel values.
 16. The imaging system ofclaim 12, wherein the ISP is further configured to perform high dynamicrange (HDR) processing of the first subset of pixel values.
 17. Theimaging system of claim 12, wherein the ISP is further configured toperform high dynamic range (HDR) processing of the second subset ofpixel values.